Deciding Indexing Strings with Statistical Analysis

Deciding indexing string is important for Information Retrieval. Ideally, the strings should be the words that represent the documents or query. Although each single word may be the first candidate of indexing strings for English corpus, it may not ideal due to the existence of compound nouns, which are often good indexing strings, and which depends on genre of corpus. The situation is even worse in Japanese or Chi-nese where the words are not separated by spaces. In this paper, we proposed a method to decide indexing strings based on statistical analysis. The novel features of our method are to make the most of the statistical measure called adaptation and not using language dependent resources such as dictionaries and stop words list. We have evaluated our method using Japanese test collection, and we have found that our method actually improves the precision of information retrieval systems.